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Abstract 

We introduce a new measure of activity of financial markets that provides a direct access to 
their level of endogeneity. This measure quantifies how much of price changes are due to endoge- 
nous feedback processes, as opposed to exogenous news. For this, we calibrate the self-excited 
conditional Poisson Hawkes model, which combines in a natural and parsimonious way exoge- 
nous influences with self-excited dynamics, to the E-mini S&P 500 futures contracts traded in 
the Chicago Mercantile Exchange from 1998 to 2010. We find that the level of endogeneity has 
increased significantly from 1998 to 2010, with only 70% in 1998 to less than 30% since 2007 of 
the price changes resulting from some revealed exogenous information. Analogous to nuclear plant 
safety concerned with avoiding "criticality" , our measure provides a direct quantification of the 
distance of the financial market to a critical state defined precisely as the limit of diverging trading 
activity in absence of any external driving. 
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I. INTRODUCTION 



Human societies and more generally biological communities are characterized by signifi- 
cant levels of interactions developing into complex network of interdependencies, leading to 
the emergence of remarkable dynamical properties and abilities. These societies are how- 
ever not just evolving spontaneously or endogenously. In absence of external "forces" , they 
would freeze, regress or die. Essential to their organization is the flux of exogenous influences 
that allow them to "feed", adapt, learn and evolve. For instance, both external stimuli and 
endogenous collective and interactive wiring between neurons are essential for brain devel- 
opment and continuous performance. Scientific discoveries result from the interplay between 
the maturation of ideas and technical progress within the scientific community as well as 
serendipity jl, 2]. One can find many other examples of the essential interplay between en- 
dogenous processes and exogenous dynamics in interesting out-of-equilibrium systems, that 
is, almost everywhere from the micro to the macro worlds jjj. 

A fundamental question is: how much of the observed dynamics is due to the external 
influences versus internal processes? Is it possible to quantify the interplay between exo- 
geneity and endogeneity? Can this be used for characterizing the robustness of systems and 
for developing diagnostics of fragility and of incoming crises as well as upside potentials? 

Here, to address these questions quantitatively, we consider financial markets as 
paradigms of complex human societies, in which external news play the role of the ex- 
ogenous influences impacting investors whose elaborate interactions via complex social and 
economic networks lead to price formation. In a nutshell, one can indeed state that financial 
markets are nothing but the engines through which information is transformed into prices. 
But, how much of the news and information are really captured by prices? Is it not the case 
that prices also reflect idiosyncratic dynamics of social networks that may lead to deviations 
from true valuation? The arguably most important question in financial economics is indeed 
to what degree are prices faithful embodiments of information? And, correlatively, what is 
"information" ? 

Financial economists have introduced the "Efficient Market Hypothesis" (EMH) that 
states, in its ideal limit, that the market absorbs in full and essentially instantaneously the 
flow of information by faithfully reflecting it in asset prices {4- 7]. The EMH amounts to 
considering the process of price formation as perfect with almost instantaneous reaction and 
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infinite precision, so that only external influences show up. In other words, according to 
the EMH, prices are just reflecting news: whatever the internal structure of the market, 
the EMH assumes that the market is sufficiently fast and effective so that its converges to 
an equilibrium price justly reflecting exogenous information, while all endogenous processes 
have had time to converge after each given exogenous shock, thus disappearing from the 
observations. As a result, markets are assumed to be driven only by an external inputs 
of information and only reflect them. Only updates of these inputs can change investors' 
anticipation and thus prices. In particular, such extreme events as financial crashes are, 
according to the EMH, the signature of exogenous negative news of large impact. 

In reality, it is now recognized that prices move much too much compared with what 
would be expected from the EMH (even corrected for the costs of gathering information), i.e., 
from the volatility of fundamental news proxied for instance by dividends [8, 9|. Moreover, 
according to the EMH, large price moves should only occur with significant geo-politico- 
economic-financial news. This prediction has been refuted by diverse studies comparing price 
movements and relevant news at the daily time scale 10] and for high frequency financial 
data [ill H], which showed that only a small fraction of price movements could be explained 
by relevant news releases. This suggests that price dynamics are mostly endogenous and 
driven by positive feedback mechanisms involving investors' anticipations that lead to sel 
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fulfilling prophecies, as described qualitatively by Soros' concept of "market reflexivity" 

The contribution of the present work is to provide what is, to the best of our knowledge, 
the first quantitative estimate of the degree of reflexivity, measured as the proportion of 
price moves due to endogenous interactions to the total number of all price moves that also 
include the impact of exogenous news. We use the self-excited conditional Poisson Hawkes 



model 



14j . which combines in a natural and parsimonious way exogenous influences with 



self-excited dynamics. According to the Hawkes model, each event, i.e. price change, may 
lead to a whole tree of offsprings, i.e. other price changes. The Hawkes model provides 
a natural set-up to describe the endogenous mechanisms resulting from herding as well as 
from strategic order splitting, which lead to long-range correlation in the series of trade 
initiations (buy-side or sell-side). The self-excited Hawkes branching process allows us to 
classify different types of volatility shocks and to separate the exogenous shocks from the 
endogenous dynamics. In particular, the Hawkes model provides a single parameter, the so- 
called "branching ratio" 'n' that measures directly the level of endogeneity. The parameter 
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n can be interpreted as the fraction of endogenous events within the whole population of 
price changes. Our observations of this measure of endogeneity reflect a robust behavioral 
trait of human beings who tend to herd more at short time scales in time of fear and panic. 
Our study thus complements the evidence for herding at the time scales of years over with 
financial bubbles develop 15|, by showing the existence of herding at short time scales 
according to a different mechanism than the ones operating at large time scales. 



II. THE DATA 



We use the E-mini S&P 500 futures contracts (ticker-symbol ES), which are traded in 
the Chicago Mercentile Exchange. Being introduced in 1997 as a supplement to the regular 
S&P 500 futures contracts with reduced contract sizes, E-mini's had attracted a lot of small 
investors and has became one of the most actively traded contracts in the world. Our 
dataset contains all transactions and changes in supply and demand (best bid and best ask 
sides) from January 5, 1998 to August 29, 2010 (in total 2'431'967'666 records including 
298'586'423 transactions), which are recorded with corresponding volume and timestamps 
that are rounded to the nearest second. The dataset was cleaned of gaps and non-trading 
days. At every moment, there are traded futures contracts with 5 different maturities in the 
March Quarterly Cycle (with expirations on the third Friday of March, June, September 
and December). However the rollover dates are 8 days before the expiration dates, i.e. on 
the second Thursdays of each of these months. At the rollover date, the liquidity (measured 
in volume) of the contract that is going to expire is switched to the contract that will expire 
at the following quarter. For the analyzed dataset, the volume of the most actively traded 
contracts accounts for 96.7% of the total traded volume (99.1% when excluding rollover 
weeks). Thus, in our studies, we have focused only on the most actively traded contracts - 
the contract whose rollover date is closest to the given moment. 

The analyzed dataset presents highly non stationary properties both at fine-grained and 
coarse-grained scales. At the daily scale, the trading activity has increased dramatically 
over these 10 years: starting in 1998 with an average of 7,000 transactions per day on 
16,000 contracts, the average daily volume in 2009 consisted in 148,000 transactions on 
approximately 1,837,000 contracts (see fig. H]A.). At the same time, during any given day, 
the activity is very low outside Regular Trading Hours (9:30-16:15 EST), and exhibits the 
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well-known the U-shape intraday seasonality, which is illustrated in fig. [3]A.. 

At any given moment t, one may distinguish four different prices in the market, that 
reflect its different properties: (i) the last transaction price Pt r (t), at which the previous 
transaction was executed, (ii) the best ask price a(t) and (iii) the best bid price b(t) at which 
market participants may immediately correspondingly by and sell an asset, and (iv) the 
mid-price, which is defined as the average of best bid and ask prices: p m (t) = (a(t) + b(t))/2. 
The bid and ask prices reflect correspondingly demand and supply of the liquidity providers; 
the transaction price reflects actions of liquidity takers, and mid-price changes result from 
actions of all market participants: both liquidity providers and takers. The transactions are 
triggered when a market order arrives. In case of a buy market order, the transaction is 
executed at the best ask price, while a sell market order triggers a transaction at the best 
bid price. Since the sequence of order arrivals is stochastic with the sign of order being a 
random variable, the last transaction price will jump from best bid to best ask price and 
back even without changes in the balance between supply and demand. This stochastic 
behavior, which is called "bid-ask bounce", represents a kind of "noise source" to the price. 

The idea that the last transaction price in high frequency financial data is a poor proxy of 
the unobservable asset's value, which is subjected to the additive "microstructure noise", is 
a well established concept in the market microstructure literature (see for instance, 161, and 

n 

the concept of "noise traders" by F. Black [17]). In contrast to the last transaction price, 
the mid-price is free from the bid-ask-bounce and is changed only when the balance between 
supply (liquidity providers) and demand (liquidity takers) is upset. Therefore, the mid-price 
is claimed to be a better proxy for the asset value, given the information available 18|, ll9j. In 
the "market impact" (or "price impact" ) literature that study the question of how much the 
price of an asset will change due to a single market order execution, the mid-price became 
the "default measure" of the price movements (see, for instance, extensive review 201] ). 

In the present study, we consider the changes of the mid-price of E-Mini S&P 500 futures 
as the best proxy for market movements as a whole. More precisely, we apply and test 
the Hawkes model to events corresponding to changes in mid-price of E-mini's within the 
Regular Trading Hours (in total 24'309'652 events). 
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III. SELF-EXCITED HAWKES MODEL 



A class of models based on continuous or discrete stochastic price processes have built 



on Bachelier's random walk model 



21] to incorporate stylized facts, such as quasi-absence 



of linear autocorrelation of returns, the existence of long-memory in the volatility, volatility 
clustering and multifractality, fat tails in the distributions of returns, correlation between 
volatility and volume, time-reversal asymmetry, the leverage effect, gain-loss asymmetries 
and others (see for instance 
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23|). 



Another class of models view the price formation process as discontinuous, i.e., following 
point processes reflecting the discrete nature of market order arrivals. The Poisson point 
process is the simplest of this class, in which events occur independently of one another 
with a constant arrival rate A. Having no correlation structure, the Poisson point process 
cannot describe the stylized facts of real order flows, such as (i) clustering of order arrivals, 
(ii) long memory in inter-trade intervals 
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25l | . (iii) slower-than-exponential decay of the 



distribution of inter-trade intervals 
trades 20] and (v) multifractal scaling of inter-trade intervals 



271 ] . (iv) long memory of the signs of successive 
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291 ] . A first attempt 



to account partially for these stylized facts characterizing high frequency transaction data 
uses a class of self-excited processes called Autoregressive Conditional Durations (ACD) 



model 
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31] . which describes the inter-event durations with a GARCH-type equation. A 
more consistent approach based on t he g eneralization of the self-excited Hawkes model [ijj 
was developed in the working paper 32] (published later with corrections 33]), where the 
author used a bivariate Hawkes process to model the arrival times of market buy and sell 
orders. This approach later became the "gold standard" for the use of self-excited models to 



describe high frequency order flows 



M 



35] and was later extended to account for the actions 



of liquidity providers in the construction process of the order book 36M38] . Recently, the 



multivariate Hawkes process was used to model last transaction price data and, in particular, 



the signature plot and the Epps effect [39|, l40 |. 



The Hawkes point process can be regarded as the generalization of the non-homogeneous 
Poisson process, whose intensity A(t) (defined such that \(t)dt is the expected value of the 
number of events in the time interval [t, t + dt)) not only depends on time t but also on the 
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history of the process according to 

\t(t) = (i(t) + ^h(t-ti) , (1) 

where ij are the timestamps of the events of the process, fi(t) is a background intensity 
that accounts for exogenous events (not dependent on history) and h(t) is a memory kernel 
function that weights how much past events influence the generation of future events and 
thus controls the amplitude of the endogenous feedback mechanism. For our purposes, the 
Hawkes process presents two interesting properties. First, the external influences on the 
system (//(£)) and the internal feedback mechanisms (h(t)) can be clearly isolated in their 
linear additive contributions to the conditional intensity A t (t) in expression ([1]). Second, the 
linear structure \t{t) in the Hawkes model allows one to map it exactly onto a branching 
process [4l|]. This implies that the process exhibits a critical bifurcation controlled by the 
so-called branching ratio parameter to be defined shortly below. 

In the language of branching processes, all events belong to one of two classes — im- 
migrants (or using earthquake terminology, main events) and descendants (or aftershocks) . 
The exogenous immigration (described by the background intensity //(£)) triggers clusters 
of descendants. Namely, every zeroth order event (immigrant) can trigger one or more 
first-order events, each of whom in turn can trigger several second-order events and so on 
over many generations (see fig. [T|). All first-, second- and higher order events (descendants) 
form the cluster of aftershocks of the main event as a result of the self-excited (endogenous) 
generating mechanism of the system. 

The crucial parameter of the branching process is the so-called branching ratio (n), which 
is defined as the average number of daughter events per mother event. There are three 
regimes: (i) sub-critical (n < 1), (ii) critical (n = 1) and (iii) super- critical or explosive 
(n > 1). Starting from a single mother event (or immigrant) at time t\, the process dies 
out with probability 1 in the sub-critical and critical regimes and has a finite probability 
to explode to an infinite number of events in the super-critical regime. The critical regime 
for n = 1 separates the two main regimes and is characterized by power law statistics of 



the number of events and in the number of generations before extinction |42|]. For n < 1, 
the process is stationary in the presence of a Poissonian or more generally stationary flux of 
immigrants. 

In the sub-critical regime, in the case of a constant background intensity — n — 



const), the branching ratio is equal to the proportion of the average number of descendants 



in the whole population 



43]. In other words, the branching ratio is equal to the proportion of 



the average number of endogenously generated events among all events. To see this, since by 
definition n is the average number of first-generation events per immigrant, the total average 
number of events of all generations triggered by a given mother event is n + n 2 + n 3 + .... = 
n/ (1 — n). The fraction of such triggered events to the total number of events (that therefore 
includes the mother event) is thus the ratio of n/(l — n) to 1 +n/(l — n), which is equal 
to n. Using the branching property that the generations of events associated to different 
mother events are forming independent "branches" , the ratio n holds for each cluster so that 
the average ratio of all triggered events by all immigrants to the total number of events is 
equal to n. Calibrating n on the data therefore provides a direct quantitative estimate of 
the degree of endogeneity. 

There are several routes to determine n. One is to reverse-engineer the clusters and 
calculate the average number of direct descendants to any given event. This can be done 
via the stochastic declustering (parametric 44j and non-parametric 45]) method, which 
amounts to reconstruct from the sequence of events (fig. [1] bottom) the original cluster 
structure (fig. [1] top), or at least distinguish between descendants and immigrants, but this 
may have severe limitations 46] . A simpler way is to just use the definition 



n 



h(t)dt . 



(2) 



Given the parametric form of the kernel h(t;9), one can estimate the parameters 6 for 
the given realization of the process t\, . . . , tjv, using the Maximum Likelihood Estimation 
method, which benefits from the fact that the log-likelihood function is known in closed 



form for Hawkes processes 
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48] 



IV. CALIBRATION TO MID-PRICE CHANGES DATA AND GOODNESS OF 
FIT ANALYSIS 



For the present application, we use the classical model ([1]) and assume (i) a constant 
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40j: h{t) 



background intensity fi(t) = const with (ii) an exponential kernel [32, 
ae _/3 *£(t), where /3 > and is the Heaviside function that ensures causality. Given this 
parametrization, the branching ratio ([2]) is given by n = a/ (3. Choosing n as an independent 
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parameter and substituting it into expression (pQ), we finally write the conditional intensity 



where ij are timestamps of the individual events and fi, n, (3 are parameters of the model. In 
this formulation, all events (changes of mid-price) have identical impact on the conditional 
intensity, independently of the direction and size of mid-price changes or volumes at the 
bid or ask sides. This simplification can be removed by considering the marked Hawkes 
process, with marks being functions of the volumes or increments of prices. However, residual 
analysis indicates that the selected model ([3]) fits the data with an excellent precision, and 
the extension to marked or multivariate Hawkes processes is not necessary. 

The assumption that the background intensity is constant implies for n < 1 that the 
events time series is stationary. As pointed earlier, the activity (the flow of events) is non 
stationary and, in particular, subjected to intraday seasonality. To address this issue, one 
needs to consider the smallest possible intervals. But, decreasing the size of the intervals 
decreases the number of events that are used for the estimation and thus its robustness. 
More important, the size of the time window limits the memory of the endogenous process 
taken into account in the estimation procedure. In other words, considering time intervals of 
just a few minutes prevents capturing the memory effects that are developing over the scale 
of hours. In the present work, we resolve this trade-off by considering time intervals of a few 
tens of minutes (namely, windows of 10, 20 and 30 minute). In such short time intervals, 
the parameters of the Hawkes model ([3]) can be considered approximately constant. At the 
same time, these intervals are wide enough to capture the endogenous memory of the system 
due to the algorithmic and high-frequency trading that operate at the scales of seconds to 
milliseconds. Moreover, windows of 10 minutes or more contain typically more than 100 
events, which allows a reliable calibration (there were on average 150 events per 10 minutes 
in 1998, 350 in 2010 and 890 in 2008). 

Another characteristic of the data needs to be addressed before calibration. Due to 
timestamps being rounded to the nearest second, the dataset contains multiple events with 
equal timestamps. On average, there are approximately 0.26 events (mid-price changes) per 
second in 1998, and 1.5 events per second in 2008. However, during the so-called "Flash- 
crash" event (May 6, 2010 14:45 EST), there was a peak of 194 mid-price changes per second. 
The Flash-crash event occurred as algorithmic and high-frequency traders in S&P 500 E- 




(3) 
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mini futures contracts triggered a dramatic fall in other markets 50] (see fig. [5]). To address 
this issue of multiple events per timestamp which may bias the calibration of the model, we 
round the timestamps by randomly redistributing events with the same timestamps within 
each second interval. This amounts to assuming that each event occurring within one second 
is independent of all the others within the same second interval (but not between different 
seconds). To verify that this procedure does not bias the calibration of the Hawkes model, 
we have tested it on synthetic time series obtained by numerical simulation of the Hawkes 
process (j3J) with parameters (/i, n, (3) close to the calibrated values on the real data. Our 
simulations show that the estimation errors, which result from the finite sample size and the 
rounding of timestamps, are low (standard deviations of 1.4% for /i, 0.6% for n and 3.5% for 
(3). Additionally, after the estimation of the parameters of the real time series, we performed 
50 bootstraps obtained by 50 different realizations of the randomization within each second in 
each time- window in 1998-2010, allowing us to study error bars of the estimations of (/i, n, 
The standard deviation of the estimated parameters is relatively small and decreases in time 
with increase of the number of events per time window. For instance, the standard deviation 
of the estimation of n was approximately 0.13-0.18 (in absolute value) in 1998, 0.05-0.07 in 
2000 and after 2002 it never exceeded the level of 0.035, decreasing by 2010 to the range of 
0.005-0.015, implying a l%-2% estimation error. 

The standard qua ntification of the goodness-of-fit of the data by the Hawkes process uses 
residual analysis [49], which consists in studying the so-called residual process defined as 
the nonparametric transformation of the initial time-series tj into ^ = \t(t)dt, where 
At(t) is the conditional intensity of the Hawkes process (j3j) estimated with the maximum 
likelihood method. Under the null hypothesis that the data has been generated by the 
Hawkes process ([3]), the residual process £j should be Poisson with unit intensity [52]. Visual 
cusum plot and Q-Q plot analysis show the excellent explanatory power of the Hawkes 
model with respect to the data. Visual analysis was complemented with rigorous statistical 
tests. Under the null hypothesis of agreement of data with the model (Poisson statistics 
of the residual process the inter-event times in the residual process Aj = & — £j_i 
should be exponentially distributed with CDF F(A) = 1 — exp(— A). Thus, the random 
variables Ui = F(Ai) = 1 — exp(— Aj) should be uniformly distributed in [0,1]. We have 
performed rigorous Kolmogorov-Smirnov tests for uniformity for each calibration of the 
Hawkes process over 1998-2010 in moving time windows of 10 minutes (198'713 estimates), 
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20 minutes (193'877 estimates) and 30 minutes (188'909 estimates) that are swept through 
the Regular Trading Hours with a step of 5 minutes. We rejected the model in a given time 
interval if the hypothesis of a uniform distribution of the residuals C/j could be rejected at 
the 5% confidence level for each of the 50 different estimations obtained with bootstrapping 
performed for that time interval. With this criterion, we have found that, out of all 10 
minutes intervals, 240 intervals (0.12%) could not be described with the Hawkes model (J3]) 
with exponential kernel. For 20 minutes intervals, we could reject 3T80 calibrations (1.64%) 
and, for 30 minutes intervals, 23'878 calibrations (12.64%) could be rejected. Fig. [2] presents 
the empirical cumulative distribution of the maximal p-values for estimations performed in 
windows of different sizes. The larger number of rejected calibrations obtained for longer 
time windows results from fits that are generally poorer due to intraday non-stationarity of 
the data. Extending the time window size to one hour, we find that more than 50% of all 
calibrations are rejected at the 5% confidence level. Working with these short time intervals 
of 10 to 30 minutes, we exclude from our following analysis the small fractions of windows 
for which the data could not be described with the model ([3]) at the showing threshold of 
5%. Overall, different tests presented in this section confirm that (i) the chosen exponential 
kernel ([3]) describes the data well, (ii) the random redistribution of the timestamps within 
each second does not affect the results of the estimation procedure and (iii) the assumption 
that the parameters are constant in each window is valid. 



V. BRANCHING RATIO n AND LEVEL OF ENDOGENEITY IN US FINANCIAL 
MARKETS 



To analyze the short-term reflexivity of the market, we use the maximum likelihood 



estimator 
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48] to calibrate the Hawkes model ([3]) in time windows of 10, 20 and 30 



minutes spanning every day from 1998 to 2010 with a 5 minutes time step. We excluded 
the days when trading was closed before 16:15 EST or with nonactive trading. We also 
filtered out the trading days with daily volume less than the 5% quantile of daily volumes 
for each given year. Fig. [3] illustrate an example of the intraday behavior of the parameters 
in March 2009. Panel [3]A. illustrates the U-shape of the intraday seasonality in the trading 
activity discussed above: one can observe that the number of transactions and the number 
of mid-price changes (as well as trading volume and the numbers of market and limit orders 
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not presented in the figure) drop by almost 50% over the lunch time in comparison with 
the opening and closing levels. This seasonality is observed in the estimated background 
intensity fi of exogenous events in the market (panel EB). In contrast to the background 
intensity, the branching ratio (panel EP) that captures the endogenous impact of reflexivity 
does not exhibit a U-shape. As one can see from the figure, the branching ratio fluctuates 
around some mean value, showing sometimes non-regular excursions that will be investigated 
further below in two case-studies discussed in section IVT1 While the average intraday pattern 
of the branching ratio can be considered as approximately constant in a first approximation, 
its shape slightly varies from year to year. We attribute this effect to the evolution of the 
trading algorithms that are the main source of the short-term reflexivity measured by n. 
Finally, we stress the good coincidence of results obtained from the estimations performed 
in windows of different sizes, which supports that the proposed method is robust. 

To analyze the evolution of the parameters over the whole period of 1998-2010, we have 
averaged the estimates for the parameters (/i, n, 0) over all windows within a two month 
period. The curves shown in fig. H] give the average values of the parameters as a function 
of the middle time t over the time intervals [t — A m ,t + A m ], where A m = 1 month. We 
also determined the quantiles over the set of all 10, 20 and 30 minutes windows in each two 
month interval. Fig. HP and D show that the results obtained for the 10, 20 and 30 minutes 
windows are practically undistinguishable. This observation together with the narrowness 
of the 10%-90% quantile range, especially for the estimation of the branching ratio, confirm 
both the status of the Hawkes model as excellent description of the data and the robustness 
of our estimation procedure. 

Comparison of Fig. HJ\-C shows that (a) the number of mid-price changes (panel A), (b) 
the daily volatility (panel B) and (c) the background intensity [i (panel C) are behaving 
similarly with coincident major peaks associated with major phases of market instabilities, 
during and following the burst of the ICT dotcom bubble 53J and associated with the 
financial crisis that started in 2007 and culminated with Lehman Brothers bankruptcy [54]. 
Note that the increase of trading activity from 1998 to 2010, as proxied by volume (Fig. SJA.) 
is not accompanied by an increase of the background intensity fi of exogenous events in the 
market. This makes intuitive sense since /i should reflect the genuine news impacting the 
market. 

In contrast, the time evolution of the branching ratio n presented in fig. HP exhibits a 
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very different behavior. The first important observation is that, since 2002, n has been 
consistently above 0.6 and, since 2007, between 0.7 and 0.8 with spikes at 0.9. These values 
translate directly into the conclusion that, since 2007, more than 70% of the price moves are 
endogenous in nature, i.e., are not due to exogenous news but result from positive feedbacks 
from past price moves. 

It should be emphasized that the increase of the branching ratio over the period 1998- 
2010 is not due to an increase of the trading activity (measured in number of transactions 
or volume) over the same period. Neither transactions, nor volume enter directly into the 
formulation of the model, since individual transactions do not necessary result in a change 
of the mid-price. As a simple example, doubling the number of transactions by splitting 
each of them into two independent transactions (to keep the daily volume constant) does not 
affect the dynamics of the mid-price at all. Similarly, keeping the number of transactions 
constant and doubling the volume of each of them (doubling the volume of each incoming 
market order) while simultaneously doubling the volume of all incoming limit orders again 
would not change the dynamics of the mid-price. The "decoupling" of market transactions 
from the simple measure of activity can be seen from fig. H]A., where the dramatic increase 
of volume in 1998-2007 was not accompanied by an increase of the number of mid-price 
changes. In fact, the latter decreased in 2003-2005. 

In order to reject the possibility that the observed increase of the branching ratio over 
1998-2010 is due to the increase of the trading activity over that period, we have performed 
the following test. We have fixed the number of mid-price changes per day, but have redis- 
tributed these events such that, within one day, their dynamics was described by a Poisson 
process. This "reshuffling" of the time series amounts to keeping the price trajectories, the 
daily volume, the number of price and mid-price changes per day unchanged (i.e., keep- 
ing the same trajectories as shown in fig. H]A. and B), while distorting time such that the 
intervals between consecutive mid-price changes within one day become uncorrelated and 
exponentially distributed. We have then performed exactly the same procedure as described 
above (dividing each day in 10 minutes interval and estimating the parameters of the Hawkes 
process ([3])) within these intervals. As one can see from the bottom shaded area in fig. HP 
that represents the 90% quantile of the branching ratio estimated within such distorted time 
series, one can clearly reject the hypothesis that the branching ratio is sensitive to, or equiv- 
alently provides another measure, of trading activity. This quantitative result supports the 
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key property of the Hawkes model, which is that the branching ratio is not determined by 
the average rate of events but by the degree of self-excitation of the system. 

In addition to the increase of the level of endogeneity in 1998-2010, one can observe 
another remarkable fact in Fig. HP corresponding to the existence of four market regimes 
over the period 1998-2010: 

(i) In the period from Q 1-1998 to Q2-2000, the final run-up of the dotcom bubble is 
associated with a stationary branching ratio n fluctuation around 0.3. 

(ii) From Q3-2000 to Q3-2002, n increases from 0.3 to 0.6. This regime corresponds to the 
succession of rallies and panics that characterized the aftermath of the burst of the 

nn 

dot-com bubble and an economic recession [56|, |57 |. 

(iii) From Q4-2002 to Q4-2006, one can observe a slow increase of n from 0.6 to 0.7. This 
period corresponds to the "glorious years" of the twin real-estate bubble, financial 



product CDO and CDS bubbles, stock market bubble and commodity bubbles [54 1. 



(iv) After Ql-2007 the branching ratio stabilized between 0.7 and 0.8 corresponding to the 
start of the problems of the subprime financial crisis (first alert in Feb. 2007), whose 
aftershocks are still resonating at the time of writing. 

As already mentioned, the value of the branching ratio larger than 0.7 since 2007 indi- 
cates that more than 70% of the price movements can be attributed to endogenous processes 
occurring in the market. Notwithstanding such rather large calibrated branching ratios n, 
they are most likely underestimations of the real values. Indeed, the Hawkes self-excited 
model has been calibrated on short intra-day time windows, using a short-memory expo- 
nential kernel h(t). This means that price moves before any given time window of 10, 20 or 
30 minutes that could trigger price changes within the window are not taken into account. 
This truncation is known to decrease artificially the observed branching ratio n and increase 



the background rate \x 58|. In other words, neglecting past events before the short-term 
windows and their triggering effect leads to the misattribution that many of the endogenous 
events are exogenous. 

The endogeneity of n > 70% that we observe in the short time windows (10, 20 and 
30 minutes) captures short-term feedback mechanisms within financial systems that can be 
interpreted as short-term reflexivity of humans and algorithmic trading systems. Indeed, 
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the growth of the branching ratio n coincides with the appearance and dramatic growth of 
algorithmic and high-frequency trading, whose birth is usually dated to 1998 when the U.S. 
Securities and Exchange Commission (SEC) authorized electronic exchanges. In the early 
2000s, high-frequency trading (defined as the high speed component of algorithmic trading) 
was quite rare and accounted for less than 10% of all equity orders. In subsequent years, its 



importance grew rapidly, increasing by about 164% between 2005 and 2009 59|. In 2009, 
the proportion of hig 
by the TABB Group 



the proportion of highfcquency trading in US markets was estimated as more than 60% 



60] and the Aite Group [6l|. Thus, there is no contradiction between 
a rather low value n « 0.3 during the final run-up of the dotcom bubble from Ql-1998 to 
Q2-2000 and the strong herding that is often invoked to explain its development. As said 
above, our calibration refers to short-term endogeneity at the time scale of 10 minutes, while 



the herding mechanism thoug 



at time scales of years [15|, |62 |. 



it to be at the origin of the dotcom bubble has been operating 



VI. DIAGNOSTIC OF CRITIC ALITY: TOWARDS THE FORECAST OF FLASH 
CRASHES 

We now show with the famous example of the "flash-crash" of May 6, 2010 that the 
calibration of the level of endogeneity n may provide instantaneous characteristic signatures 
of anomalous market regimes, which could be used as precursors for forecasts. 

For this, we consider two extreme events that occurred in the spring of 2010 that are 
comparable in their price amplitudes and market-wide impacts, as can be seen from the top 
four panels of Fig. |5j On April 27, 2010, all major US indices and stocks felt significantly 
after Standard&Poors cut Greece's debt rating to "BB+" and lowered Portugal's debt rating, 



raising fears that a euro zone debt crisis could slow the global economic recovery [5JJ. On 
May 6, 2010, in a general atmosphere of pessimism and worry concerning the debt crisis in 
Greece, a large market order triggered a flurry of activities by algorithmic traders resulting in 
a large drop of the price of S&P 500 E-mini futures. Due to the coupling with other markets 
via hedging and portfolio effects, this drop cascaded to many other markets, triggering 
drawdowns of up to 60% in some of them [50 1. 

The top four panels of Fig. |5] show that the two extreme events of April 27 and May 
6, 2010 have similar price drops and volume of transactions. In particular, we find that 
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the volume was multiplied by 4.7 for April 27, 2010 and by 5.3 times for May 6, 2010 in 
comparison with the 95% quantile of the previous days' volume. The main difference lies 
in the trading rates and in the branching ratio. Indeed, the event of April 27, 2010 can 
be classified according to our calibration of the Hawkes model as a pure exogenous event, 
since the branching ratio n (fig. EPi) does not exhibit any statistically significant change 
compared with previous and later periods. In contrast, for the May 6, 2010 flash crash, one 
can observe a statistically significant increase of the level of endogeneity n (fig. EP2). At the 
peak, n reaches 95% from a previous average level of 72%, which means that, at the peak 
(14:45 EST), more than 95% of the trading was due to endogenous triggering effects rather 
than genuine news. 

Comparing the trading rates in panels Ci and C2 of Fig. El one can observe that until 
time 14:30 EST the trading rate of May 6 was increasing at a similar rate as for April 
27. Therefore, on the basis of trading rates, it would not have been possible to predict the 
subsequent jump in trading rate that occurred during the flash crash. But the panel D 2 for 
n shows that, at this time, the instantaneously statistical estimation of the branching ratio n 
was already giving an abnormal reading, in the sense that n jumped above the 95% quantile. 
In contrast, for the April 27, 2010 event, the branching ratio was fluctuating normally within 
its normal band. 

This comparison suggests that the estimation of the branching ratio provides a novel pow- 
erful metric of endogeneity, which is much richer than standard direct measures of activity 
such as volume and trading rates. Indeed, the branching ratio provides a direct access to 
the level of endogeneity of the market. The distance of n to the critical value 1 can be taken 
as a gauge of the degree to which the market is going "critical" . The term actually is more 
than just suggestive: "criticality" in nuclear reactions precisely refers to a branching process 
of neutrons triggering and being created by nuclear reactions for which the process does not 
stop but may in fact explode. Similarly, as n approaches 1, we can state that the market 
approaches "criticality" in this precise sense of a theoretically diverging trading activity in 
absence of any external driving. 
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VII. CONCLUSION 



We have provided what is, to the best of our knowledge, the first quantitative estimate 
of the degree of reflexivity, measured as the proportion n of price moves due to endogenous 
interactions to the total number of all price moves that also include the impact of exogenous 
news. For this, we have used the self-excited conditional Poisson Hawkes model 14(, which 



combines in a natural and parsimonious way exogenous influences with self-excited dynamics. 
Within the Hawkes model framework, the parameter n takes the simple meaning of being 
the average branching ratio or, equivalently, the average number of triggered events of first 
generation per exogenous source. We have calibrated the Hawkes model to the E-mini S&P 
500 futures contracts traded in the Chicago Mercantile Exchange from 1998 to 2010. We 
find that the level of endogeneity has increased significantly from 1998 to 2010, with only 
70% in 1998 to less than 30% since 2007 of the price changes resulting from some revealed 
exogenous information. We have also documented a drastic difference in the change of n 
before and during two extraordinary flash crash events that occurred respectively on April 
27, 2010 and on May 6, 2010. For the former, we find that the branching ratio n remained 
constant, exemplifying the exogenous nature of the crash. For the later, in contrast, we 
document both a precursory early rise of n followed by strong increase culminating very 
close to the critical value n = 1, suggesting a strong endogenous component. 

In conclusion, the present study enlarges considerably the usefulness and operational 
implementation of the Hawkes model, which was already used in the study of the dynamics 



of book sales 



63j and of views of YouTube videos [64|. Indeed, these two previous studies 



have selected the blockbusters, which are characterized by large social and/or attention. This 
amounts to introducing a selection bias towards dynamics (i.e. those books and videos) that 
are close to critical (n ~ 1), as confirmed by the pure power law behavior 65] documented 



in 



63 



64|. In contrast, the present study has demonstrated how to quantify the level of 



endogeneity n, which is found to characterize different social regimes. Our work opens the 
road towards the full utilization of the dynamics of n in order to diagnose different regimes 
and to possibly forecast impending crises associated with the approach to criticality n > 1. 
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FIG. 1: (Color online) Illustration of the branching structure of the Hawkes process (top) and 
events on the time axis (bottom). Different colors of markers correspond to different clusters, the 
dashed lines denote descendants of the same cluster and the numbers next to each event denotes 
its order within the cluster. This picture corresponds to a branching ratio equal to n = 0.88. 
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p-value 

FIG. 2: Cumulative distribution of the p-values obtained in the test for uniform distribution (see 
text) used to estimate the Hawkes model, performed in windows of (a) 10 minutes, (b) 20 minutes 
and (c) 30 minutes. The vertical dashed line denotes the 5% threshold at which the statistical test 
performed on the calibration of the model in a given window leads to its rejection. 



24 





10000r 


CO 


5000 - 


c 




CD 
> 


2500- 


CD 




M — 

O 


1000- 


CD 




JQ 

E 


500- 


13 
2 


250- 



0.6 
0.4 
0.2 




0.9 



0.8 








1 


























I 




1 







0.6 



09:30 10:30 11:30 12:30 13:30 

Time, EST 



14:30 15:30 



FIG. 3: (Color online) (A) Number of mid-price changes (dots) and transactions (circles) for the 
E-mini S&P 500 futures contracts in time windows of 10 minutes during the Regular Trading 
Hours in March 2009 (logarithmic scale on vertical axis). Each point represents the averaged value 
over the period from March 1, 2009 to March 31, 2009, and the shaded area correspond to the 
10%-90% quantile range obtained with the same one month of data. (B) Estimated background 
intensity (ft) and (C) branching ratio (n) of the flow of mid-price changes of the E-mini S&P 500 
futures contracts over the Regular Trading Hours. Each point at a given time t represents an 
estimate in windows of 10 (dots), 20 (squares) and 30 (crosses) minutes averaged over the period 
of 1 month. The shaded area corresponds to 10%-90% quantile range obtained with the same 1 
month of estimates for 10 minutes time windows. 
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FIG. 4: (Color online) (A) Number of mid-price changes (in millions of events) and volume (in 
millions of contracts) for the E-mini S&P 500 futures contracts over the period 1998-2010. Each 
point at a given time t represents the total volume and number of events over a two month 
interval centered on that time t. (B) Daily closing price and daily volatility estimated with the 



Garman&Klass estimator [55(. (C) Estimated background intensity (/}) and (D) branching ratio 
(n) of the flow of mid-price changes of the E-mini S&P 500 futures contracts over the period 1998- 
2010. Each point at a given time t represents an averaged over a two month interval centered on 
that time t of windows of 10 (dots), 20 (squares) and 30 (crosses) minutes (the corresponding lines 
are almost indistinguishable). The shaded area corresponds to 10%-90% quantile range obtained 
with the same 2 months of daily estimates for 10 minutes time windows. The horizontal shaded area 
in the inset (D) corresponds to the 90% quantile of estimations performed with the "reshuffled" 
time series (see text). Dots correspond to the median value and circles to arithmetic mean. 
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FIG. 5: (Color online) Case study of two extreme events that occurred during the spring of 2010. 
The left column corresponds to April 27, when almost all US markets fell significantly following the 
dramatic decrease of the credit rating of Greece and Portugal 5l|. The right column corresponds 



to May 6, 2010, the so-called "flash-crash" event, when the activity of high-frequency traders of 
the S&P 500 E-mini futures contracts leaded to a dramatic fall in other markets 50j]. Panels from 
top to bottom: Minute-by-minute price (A) and volume (B) of S&P 500 E-mini futures contracts 
(maturity: June, 2010). (C) Average number of mid-price changes per second (averaged in 10 
minutes interval). (D) Branching ratio n estimated within each 10-minutes interval. The blue 
horizontal regions depict the 5%-95% quantile intervals of the corresponding values calculated 
over the previous trading day. The pink vertical regions highlight the time periods of the most 
active trading. 
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